Position Description – Senior Databricks Engineer

Location: Sydney (Martins Place)

Clearance: Baseline (Minimum)
Start: ASAP
Contract Duration: 12 Months

Role summary

Cleared are partnering with a fincial regulator who are seeking a talented Senior Databricks Engineer.

The senior Databricks engineer will lead a team of engineers in the rollout and operational management of the Databricks platform on Azure.

Initially engaged within Program GUIDE, this role will work alongside other project roles in the design and implementation of prioritised use cases focussed on the modernisation of existing data assets using Databricks as a platform. In addition, this role will be responsible for implementing sustainable operational processes, extending job orchestration frameworks and the ongoing training and mentoring of BAU data engineers.

This role will play a critical part in the company’s transition to a modern data architecture.

Organisational environment

This role resides within the Technology team and works closely with team members across all streams of Program GUIDE.

Key accountabilities and outcomes

• Build and maintain data ingestion and transformation pipelines in Azure Databricks.

• Add new data sources, implement business rules, and produce new metrics to provide users with greater insights, supporting key business decisions.

• Work closely with industry teams to develop new data model data collections, following best practices in Kimball dimensional modelling.

• Use vendor tools to monitor and improve the data quality and security in the Databricks lakehouse.

• Extend design principles, orchestration frameworks and other platform supports where necessary.

• Serve as a data domain expert with an in-depth understanding of company’s target state architecture.

• Maintain, monitor and troubleshoot existing Databricks pipelines and environments to ensure reliability and performance.

• Mentor and lead other data engineers in the development and implementation of leading software engineering practices.

Desired experience and capabilities

• Bachelor’s degree in computer science, Information Systems, Business Administration, or a related field.

• 3+ years hands-on experience with Databricks on Azure.

• Experience with Azure cloud technologies including Azure Data Lake, Azure Blob storage and Azure Data Factory.

• Strong understanding of data architecture principles, including data modelling, ETL processes, data governance, data quality, privacy and security controls.

• Proficiency in SQL, data analysis tools, and familiarity with programming languages such as Python

• Excellent analytical and problem-solving skills, with the ability to assess complex systems and propose actionable solutions.

• Knowledge of the Medallion Architecture or similar data frameworks.

Cleared Recruitment is leading the way for Defence recruitment services, bringing together some of Australia's best talent in ICT, Engineering and business and connecting those with specialised and future thinking Defence industry suppliers to ensure Australia is mission-ready at all times.